Atmospheric Sciences and Climate Applications Using Hdf and Hdf5

نویسندگان

  • MuQun Yang
  • Robert E. McGrath
  • Mike Folk
چکیده

Since 1998, NCSA has developed a more general and robust data format, called HDF5, which was designed to support the future demands of Earth Science, such as large data storage, performance, and flexibility. The NCSA HDF group encourages applications to use HDF5 for better performance and maintenance. See extend abstract P2.42 for more information about HDF5. In summary, HDF5 has the following features that make its use attractive for many important atmospheric science applications. 1) Flexible data model 2) Open-source, free software 3) Portability 4) Emphasis on performance • MPIIO support • Improved sub-setting • In-memory compression • Alternative storage models

برای دانلود رایگان متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید

ثبت نام

اگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید

منابع مشابه

Merging the NetCDF and HDF5 Libraries to Achieve Gains in Performance and Interoperability

This proposal seeks to advance the technology available for accessing scientific data and for integrating observations with model representations in multiple disciplines by extending and merging the Network Common Data Form (netCDF [8]) library and the Hierarchical Data Format-5 (HDF5 [10]) library. These two independently developed data-access libraries and formats provide simple, efficient, a...

متن کامل

Hdf 5 – a High Performance Data Format for Earth Science

HDF is a set of data formats and software libraries for storing scientific data with an emphasis on standards, storage and I/O efficiency. HDF software is open-source and free. HDF4 is based on the original 1988 version of HDF and is backwardly compatible with all earlier versions. HDF4 file size cannot be greater than 2GB and the number of objects inside an HDF4 file must be less than 20,000. ...

متن کامل

The Impact of the Data Archiving File Format on Scientific Computing and Performance of Image Processing Algorithms in MATLAB Using Large HDF5 and XML Multimodal and Hyperspectral Data Sets

Scientists require the ability to effortlessly share and process data collected and stored on a variety of computer platforms in specialized data storage formats. Experiments often generate large amounts of raw and corrected data and metadata, which describes and characterizes the raw data. Scientific teams and groups develop many formats and tools for internal use for specialized users with pa...

متن کامل

1 NetCDF - 4 Performance Report

The netCDF-4 project has generated enormous interest in both the netCDF and HDF user communities, and substantial reductions in data storage space have been achieved with the netCDF-4 beta releases using the HDF5 compression options. The performance of netCDF-4 is also a critical factor for users of netCDF-3 (the latest version of netCDF) who are considering a move to netCDF-4. In particular, t...

متن کامل

Supporting XML Based High-Level Abstractions on HDF5 Datasets: A Case Study in Automatic Data Virtualization

Recently, we have been focusing on the notion of automatic data virtualization. The goal is to enable automatic creation of efficient data services to support a high-level or virtual view of the data. The application developers express the processing assuming this virtual view, whereas the data is stored in a low-level format. The compiler uses the information about the low-level layout and the...

متن کامل

ذخیره در منابع من


  با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید

عنوان ژورنال:

دوره   شماره 

صفحات  -

تاریخ انتشار 2004